Utility Optimal Scheduling in Energy Harvesting 

Networks 



Longbo Huang, Michael J. Neely 



O 

(N 

O 

CD 

Q 



U 
O 



> 
o 



X 



Abstract — In this paper, we show how to achieve close-to- 
optimal utility performance in energy harvesting networlts with 
only finite capacity energy storage devices. In these networks, 
nodes are capable of harvesting energy from the environment. 
The amount of energy that can be harvested is time varying 
and evolves according to some probability law. We develop an 
online algorithm, called the Energy-limited Scheduling Algorithm 
(ESA), which jointly manages the energy and makes power 
allocation decisions for packet transmissions. ESA only has to 
keep track of the amount of energy left at the network nodes 
and does not require any knowledge of the harvestable energy 
process. We show that ESA achieves a utility that is within 
0(e) of the optimal, for any e > 0, while ensuring that the 
network congestion and the required capacity of the energy 
storage devices are deterministically upper bounded by bounds of 
size 0(l/e). We then also develop the Modifled-ESA algorithm 
(MESA) to achieve the same 0{e) close-to-utility performance, 
with the average network congestion and the required capacity 
of the energy storage devices being only 0([log(l/e)]^). 

Index Terms — Energy Harvesting, Lyapunov Analysis, Stochas- 
tic Network, Queueing 



I. Introduction 

Recent developments in hardware design have enabled many 
general wireless networks to support themselves by harvesting 
energy from the environment. For instance, by converting me- 
chanical vibration into energy fT], by using solar panels Q, by 
utilizing thermoeletric generators ID, or by converting ambient 
radio power into energy lH . Such harvesting methods are also 
referred to as "recycling" energy fSj. This energy harvesting 
ability is crucial for many network design problems. It frees 
the network devices from having an "always on" energy source 
and provides a way of operating the network with a potentially 
infinite lifetime. These two advantages are particularly useful 
for networks that work autonomously, e.g., wireless sensor 
networks that perform monitoring tasks in dangerous fields 
161, tactical networks Q, or wireless handheld devices that 
operate over a longer period [8 J, etc. 

However, to take full advantage of the energy harvesting 
technology, efficient scheduling algorithms must consider the 
finite capacity for energy storage at each network node. In 
this paper, we consider the problem of constructing utility 
optimal scheduling algorithms in a discrete stochastic network, 
where the communication links have time-varying qualities, 
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and the nodes are powered by finite capacity energy storage 
devices but are capable of harvesting energy. Every time slot, 
the network decides how much new data to admit and how 
much power to allocate over each communication link for data 
transmission. The objective of the network is to maximize 
the aggregate traffic utility subject to the constraint that the 
average network backlog is finite, and the "energy-availability" 
constraint is met, i.e., at all time, the energy consumed is 
no more than the energy stored. We see that the "energy- 
availability" constraint greatly complicates the design of an 
efficient scheduling algorithm, due to the fact that the current 
energy expenditure decision may cause energy outage in the 
future and thus affect the future decisions. Such problems 
can in principle be formulated as dynamic programs (DP) 
and be solved optimally. However, the DP approach typically 
requires substantial statistical knowledge of the harvestable 
energy process and the channel state process, and often runs 
into the "curse-of-dimensionality" problem when the network 
size is large. 

There has been many previous works developing algorithms 
for such energy harvesting networks. ||9l develops algorithms 
for a single sensor node for achieving maximum capacity 
and minimizing delay when the rate-power curve is linear. 
flOl considers the problem of optimal power management 
for sensor nodes, under the assumption that the harvested 
energy satisfies a leaky-bucket type property. ITTI looks at the 
problem of designing energy-efficient schemes for maximizing 
the decay exponent of the queue length. 1121 develops schedul- 
ing algorithms to achieve close-to-optimal utility for energy 
harvesting networks with time varying channels. ifTsl develops 
an energy-aware routing scheme that approaches optimal as the 
network size increases. Outside the energy harvesting context, 
lfT4l considers the problem of maximizing the lifetime of a 
network with finite energy capacity and constructs a scheme 
that achieves a close-to-maximum lifetime. ifTsl and lfT6l 
develop algorithms for minimizing the time average network 
energy consumption for stochastic networks with "always on" 
energy source. However, most of the existing results focus 
on single-hop networks and often require sufficient statistical 
knowledge of the harvestable energy, and results for multihop 
networks often do not give explicit queueing bounds and do 
not provide explicitly characterizations of the needed energy 
storage capacities. 

We tackle this problem using the Lyapunov optimization 
technique developed in ifTSl and ifTTl . combined with the 
idea of weight perturbation, e.g., ijTS) and fTg]. The idea 
of this approach is to construct the algorithm based on a 
quadratic Lyapunov function, but carefully perturb the weights 



used for decision making, so as to "push" the target queue 
levels towards certain nonzero values to avoid underflow (in 
our case, the target queue levels are the energy levels at the 
nodes). Based on this idea, we construct our Energy-limited 
Scheduling Algorithm (ESA) for achieving optimal utility in 
general multihop energy harvesting networks powered by finite 
capacity energy storage devices. ESA is an online algorithm 
which makes greedy decisions every time slot without re- 
quiring any knowledge of the harvestable energy and without 
requiring any statistical knowledge of the channel qualities. 
We show that the ESA algorithm is able to achieve an average 
utility that is within 0(e) of the optimal for any e > 0, 
and only requires energy storage devices that are of 0{l/e) 
sizes. We also explicitly compute the required storage capacity 
and show that ESA also guarantees that the network backlog 
is deterministically bounded by 0(l/e). Furthermore, we 
develop the Modified-ESA algorithm (MESA) to achieve the 
same 0(e) close-to-optimal utility performance with energy 
storage devices that are only of 0([log(l/e)]^) sizes. We 
note that the approach of using perturbation in Lyapunov 
algorithms is novel. It not only allows us to resolve the energy 
outage problem easily, but also enables an easy analysis of the 
algorithm performance. 

Our paper is mostly related to the recent work lfT2l . which 
considers a similar problem. ||l2l uses a similar Lyapunov 
optimization approach (without perturbation) for algorithm 
design, and achieves a similar [0(e), 0(l/e)] utility-backlog 
performance using energy storage sizes of 0(l/e) for single- 
hop networks. Multihop networks are also considered in lfT2l . 
However, the performance bounds for multihop networks are 
given in terms of unknown parameters. In our paper, we 
compute the explicit 0(l/e) capacity requirements for the 
data buffers and energy storage devices for general multihop 
networks for achieving the 0(e) close-to-optimal utility per- 
formance. We then also develop a scheme to achieve the same 
utility performance with only 0([log(l/e)]^) energy storage 
capacities. 

Our paper is organized as follows: In Section|II]we state our 
network model and the objective. In Section |IlI] we first derive 
an upper bound on the maximum utility. Section |IV] presents 
the ESA algorithm. The [0(e), 0(l/e)] performance results of 
the ESA algorithm are presented in Section |V] for both the 
cases when the network randomness is i.i.d. and Markovian. 
We then construct the Modified-ESA algorithm (MESA) to 
achieve the same 0(e) close-to-optimal utility performance 
with only 0([log(l/e)]^) energy storage sizes in Section fVll 
Simulation results are presented in Section IVIII We conclude 
our paper in Section IVIIII 

II. The Network Model 

We consider a general interconnected network that operates 
in slotted time. The network is modeled by a directed graph 
g = (A/",/:), where Af = {l,2,...,iV} is the set of the N 
nodes in the network, and £ = {[n, m], n,m € Af} is the set 
of communication links in the network. For each node n, we 
use TVi""* to denote the set of nodes b with [n, b] G C, and use 
Nn"''^ to denote the set of nodes a with [a, n\ G C. We then 



define dmax — max„ \Mn' |to be the maximum in-degree that 
any node n E Af can have. 

A. The Traffic and Utility Model 

At every time slot, the network decides how many packets 
destined for node c to admit at node n. We call these 
traffic the commodity c data and use R^n\t) to denote the 
amount of new commodity c data admitted. We assume that 
< Bin\t) < Rmax for all n,c with some finite Rmax 
at all time. [J We assume that each commodity is associated 
with a utility function Un^{r^'^), where r"'^ is the time 
average rate of the commodity c traffic admitted into node 
n, defined as r"'^ = liminft^oo 7 Et=o^{^" '(^)}- Each 
U^\r) function is assumed to be increasing, continuously 
differentiable, and strictly concave in r with a bounded first 
derivative and Un\Q) ~ 0. We use to denote the 
maximum first derivative of u'^f^{r), i.e., = {u'^f^)'{Q) 
and denote /3 — max„^c/3"'^- 

B. The Transmission Model 

In order to deliver the data to their destinations, each node 
needs to allocate power to each link for data transmission 
at every time slot. To model the effect that the transmission 
rates typically also depend on the link conditions and that 
the link conditions may be time varying, we let S{t) be 
the network channel state, i.e., the A^-by-A^ matrix where 
the (n, m) component of S{t) denotes the channel condition 
between nodes n and m. We assume that S{t) takes values 
in some finite set S = (si, ...,sm)- In the following, we first 
assume that S{t) is i.i.d. every time slot and use tTs- to denote 
Pr{S{t) = Si). We will later extend our results to the case 
when S{t) is Markovian. At every time slot, if S{t) = Si, then 
the power allocation vector P{t) = (P[„ „j] (t), [n, m] G C), 
where P[„^m] (t) is the power allocated to link [n, m] at time t, 
must be chosen from some feasible power allocation set P^*'). 
We assume that 'p(*') is compact for all s;, and that every 
power vector in 'p(^') satisfies the constraint that for each 
node n, < J^bejvi"^ -P[n,fe](i) < Pmax for some Pr,iax < 00. 
Also, we assume that setting any P[n.m] in ^ vector P G 'P''*'' 
to zero yields another power vector that is still in V^^^\ 
Given the channel state S{t) and the power allocation vector 
P{t), the transmission rate over the link [n,m] is given by 
the rate-power function /i[„,m](^) = lJ-[n,m\{S{t),P{t)). For 
each Si, we assume that the function ^^n.m\{si,P) satisfies 
the following properties: 

Property 1: For any vectors P,P' G 'p(^^\ where P' is 
obtained by changing any single component P[„ „i] in P to 
zero, we have: 

lJ-[n,m]{Si,P) < M[n,m](si,-P') + <5^[n,m] , (1) 

for some finite constant 6 > 0. 

'Note that this setting implicitly assumes that nodes always have packets 
to admit. The case when the number of packets available is random can also 
be incorporated into our model and solved by introducing auxiliary variables, 
as in (20). 



Property 2: If P' is obtained by setting the entry P[n.b] in 
P to zero, then: 

^J■[a,nl](^'^^^) - M[a,m](Sj,-P')) V [a, m] ^ [n,b]. (2) 

Property [T| states that the rate obtained over a link [?? , m] is 
upper bounded by some linear function of the power allocated 
to it, whereas Property |2] states that reducing the power over 
any link does not reduce the rate over any other links. We see 
that Property [T] and |2] can usually be satisfied by most rate- 
power functions, e.g., when the rate function is differentiable 
and has finite directional derivatives with respect to power 
ifTSl . and the links do not interfere with each other. 

We also assume that there exists some finite constant 
lj,max such that H[n,m]{t) < fimax for all time under any 
power allocation vector and any channel state S{t). § In the 
following, we also use M[^''f,](*) to denote the rate allocated to 
the commodity c data over link [n, b] at time t. It is easy to 
see that at any time t, we have: 

^i^^b]. (3) 

c 

C. The Energy Queue Model 

We now specify the energy model. Every node in the 
network is assumed to be powered by ?l finite capacity energy 
storage device, e.g., a battery or ultra-capacitor |[9]- We model 
such a device using an energy queue. We use the energy queue 
size at node n at time t, denoted by En{t), to measure the 
amount of the energy left in the storage device at node n at 
time t. We assume that at every time, the nodes are capable 
of tracking its current energy level En{t). In any time slot t, 
the power allocation vector P{t) must satisfy the following 
"energy-availability" constraint: 

P[nMit) < En{t), Vn. (4) 

That is, the consumed power must be no more than what is 
available. Each node in the network is assumed to be capable 
of harvesting energy from the environment, using, for instance, 
solar panels |9j. However, the amount of harvestable energy in 
a time slot is typically not fixed and varies over time. We use 
hn{t) to denote the amount of harvestable energy by node n 
at time t, and denote h{t) = {hi{t), hN{t)) the harvestable 
energy vector at time t, called the energy state. We assume 
that h{t) takes values in some finite set H — {hi, hx}, 
and that h{t) is i.i.d. over each slot. However, components in 
each hi vector may be correlated. We will later consider the 
case when h{t) is Markovian. In both cases, we assume that 
h{t) is independent of 5'(t).[l 

We let TThi = Pr{h{t) = hi). We assume that there exists 
hmax < oo such that hn{t) < hjnax for all n, t, and the energy 
harvested at time t is assumed to be available for use in time 

^Note that in our transmission model, we did not explicitly take into account 
the reception power. However, this can easily be incorporated into our model 
at the expense of more complicated notations. All the results in this paper 
will still hold in this case. 

^This is for the ease of presentation. The results in this paper still hold if 
they are correlated. 



t + I. In the following, it is convenient for us to assume that 
each energy queue has infinite capacity, and that each node 
can decide whether or not to harvest energy on each slot. We 
model this harvesting decision by using e„(t) e [0,/i„(t)] 
to denote the amount of energy that is actually harvested at 
time t. We will show later that our algorithm always harvests 
energy when the energy queue is below a finite threshold of 
size 0{l/e) and drops it otherwise, thus can be implemented 
with finite capacity storage devices. 

D. Queueing Dynamics 

Let Q{t) = {Qi^\t),n,c e J\f), t ^ 0,1,2,... be the data 
queue backlog vector in the network, where Qn\t) is the 
amount of commodity c data queued at node n. We assume 
the following queueing dynamics: 

with Q'h\0) = for all n,c e Af, Q^^^t) = Vi, and 
[a;]+ — max[a;, 0]. The inequality in (|5]i is due to the fact that 
some nodes may not have enough commodity c packets to fill 
the allocated rates. In this paper, we say that the network is 
stable if the following condition is met: 

_ 1 *-i 

Q^\imsup-YT.HQn\r)} <^. (6) 

t—^oo C 

r— n,c 

Similarly, let E{t) {En{t),n £ J\f) be the vector of the 
energy queue sizes. Due to the energy availability constraint 
dUi, we see that for each node n, the energy queue En{t) 
evolves according to the following: 

E„it + I) = E„(t) - Pln.b]it) + er,it), (7) 

with En{0) ~ for all n. |f| Note again that by using 
the queueing dynamic (|7]i, we start by assuming that each 
energy queue has infinite capacity. Later we will show that 
under our algorithms, all the En (t) values are determinstically 
upper bounded, thus we only need a finite energy capacity in 
algorithm implementation. 

E. Utility Maximization with Energy Management 

The goal of the network is thus to design a joint flow control, 
routing and scheduling, and energy management algorithm that 
at every time slot, admits the right amount of data Rn\t), 
chooses power allocation vector P{t) subject to dU, and 

^Note that we do not explicitly consider energy leakage due to the 
imperfectness of the energy storage devices. This is a valid assumption if 
the rate of energy leakage is very small compared to the amount spent in 
each time slot. 

^We can also pre-store energy in the energy queue and initialize -E„(0) to 
any finite positive value up to its capacity. The results in the paper will not 
be affected. 



transmits packets accordingly, so as to maximize the utility 
function: 

Utotir) = ^C/l^^r-), (8) 

subject to the network stability constraint (|6]l. Here r = 
(f"'^, Vn, c e TV) is the vector of the average expected admit- 
ted rates. Below, we will refer to this problem as the Utility 
Maximization with Energy Management problem (UMEM). 

F. Discussion of the Model 

Our model is quite general and can be used to model many 
networks where nodes are powered by finite capacity batteries. 
For instance, a field monitoring sensor network ||6l, or many 
mobile ad hoc networks ETI . Also, our model allows the 
harvestable energy to be correlated among network nodes. This 
is particularly useful, as in practice, nodes that are collocated 
may have similar harvestable energy conditions. 

The main difficulty in designing an optimal scheduling 
policy here is imposed by the constraint Indeed, (|4|i 
couples the current power allocation action and the future 
actions, in that a current action may cause the energy queue 
to be empty and hence block some power allocation actions in 
the future. Problems of this kind usually have to be modeled 
as dynamic programs 1221 . However, this approach typically 
requires significant statistical knowledge of the network ran- 
domness, including the channel state and the energy state. 
Another way to utilize the harvested energy efficiently is by 
developing efficient sleep-wake policies, e.g., ||23]| . Although 
our model does not consider this aspect, our algorithm can 
also be used together with given sleep-wake policies to achieve 
good utility performance in that context. 

HL Upper bounding the optimal network utility 

In this section, we first obtain an upper bound on the optimal 
utility. This upper bound will be useful for our later analysis. 
The result is presented in the following theorem, in which we 
use r* to denote the optimal solution of the UMEM problem, 
subject to the constraint that the network nodes are powered 
by finite capacity energy storage devices. The V parameter 
in the theorem can be any positive constant that is greater or 
equal to 1, and is included for our later analysis. 

Theorem 1: The optimal network utility Utotir*) satisfies 
the following: 

VUtot{r*)<^*, (9) 

where </>* is the optimal value of the following optimization 
problem: 

K 

max: <^ = F^^t?fc[/(^)(rr) (10) 

n,c k=l 

<E-^.E^i"^ E 4l^is^,pi'^\y{n,c), (11) 



E-^.E^^' E ^£.1 (12) 

K 

hi k=l 
fe=l k=l k=l 

0<rr <i?™a.,V(n,c), 
0<ej^) Wn,k,K. 

Herein = N'^+N+2.E{rl''}^^^ denotes the set of admission 
decisions used for each commodity flow. {-P^^'^jfcLi denotes 
the set of power allocation vectors that are used when S{t) = 
Si. /ij^-'^j(si, P^''*'') is the rate allocated to commodity c over 

Unk [b,n] under Si and -P^^'''. Pjfinm] power allocated 

to link [n,m\ under P^f^'^ ■ {e[^i}}k=i is the set of energy 
harvesting decisions of node n when the energy state is hi, 
and hi!^''' is the amount of harvestable energy for node n when 
h{t) = h,. 

Proof: The proof argument is similar to the one used in 
|fT9l, hence is omitted for brevity. ■ 

Note that Theorem [T| indeed holds under more general 
ergodic S{t) and h{t) processes, e.g., when S{t) and h{t) 
evolve according to some finite state irreducible and aperi- 
odic Markov chains. Also note that the objective function 
is not of the same form as Utot{')- However, it can be 
shown, using Jensen's inequality, that the optimal value of 
the above optimization problem remains the same if we push 
inside the function Un \ i.e., change the objective to 
^ Ylin c (Ea^Li ^fe^fc*^)- Below, we first have the following 
lemma regarding the dual problem of ( fTOl i: 

Lemma 1: The dual problem of dTol i is given by: 

min: g{v,v>), s.t. v 0,v> eR^ , (13) 

where v = {vn\y {n, c)), u ~ (z^„,Vn) and g{v,u) is the 
dual function defined: 

g{v,u)^ sup Y.^s.Y.^'^^.l^T.UnKrn 

-Y.^':'W''+ E ^'^"^) (14) 
-E-4 E 

Moreover, let [v* ,v*) be an optimal solution of ( fT3] l, then 
c^*<g{v\u*). 

Proof: The proof uses a similar argument as the one used 
in |fT9l . Hence is omitted for brevity. ■ 

^The number K is due to the use of Caratheodoiy's Theorem in the proof 
argument used in 1191 . 



Note that the dual function g{v, v) does not contain the terms 
'^k, Qk^\'fk^ ■ This not only simplifies the evaluation of the 
dual function, but also enables us to analyze the performance 
of our algorithm using Theorem [T] In the following, it is also 
useful to define the function gsi.hj (f , t^) for each (s^, hj) pair: 

sup IvY^Ui^Hr-') (15) 

p(si) gj^^i' I- n,c 



E-4 E 



That is, gs- hjivji^) is the dual function of ( fTOl i when there 
is a single channel state and a single energy state hj. It is 
easy to see from (fl4b and (fTSl l that: 



(16) 



Si hj 



IV. Engineering the queues 

In this section, we present our Energy-limited Scheduling 
Algorithm (ESA) for the UMEM problem. ESA is designed 
based on the Lyapunov optimization technique developed in 
lfT9l and iflTl . The idea of ESA is to construct a Lyapunov 
scheduling algorithm with perturbed weights for determining 
the energy harvesting, power allocation, routing and schedul- 
ing decisions. We will show that, by carefully perturbing the 
weights, one can ensure that whenever we allocate power to 
the links, there is always enough energy in the energy queues. 

To start, we first choose a perturbation vector 9 = (0„, n G 
Af) (to be specified later). We then define a perturbed Lya- 
punov function as follows: 

m = J E [Qi^' w]' + \ E [^"W - ^"]'- (17) 



Now denote Z{t) = {Q{t), E{t)), and define a one-slot 
conditional Lyapunov drift as follows: 



A{t) = E{L{t + 1) - L{t) I Z{t)}. 



(18) 



Here the expectation is taken over the randomness of the 
channel state and the energy state, as well as the randomness 
in choosing the data admission action, the power allocation 
action, the routing and scheduling action, and the energy 
harvesting action. We have the following lemma regarding the 
drift: 

Lemma 2: Under any feasible data admission action, power 
allocation action, routing and scheduling action, and energy 



harvesting action that can be implemented at time t, we have: 

Ait)-VE{J2u,[^HRl:'Ht))\Zit)} (19) 

71, C 

< B -VE{Y,U^HRi'\t)) \ Z{t)} 

n.c 

- (Enit) - 9n)E{ Y P[nMit) " ^nlO I Z{t)}. 

Here B N {fJ-jy^ax ~^ 2-R'max) ^[-^maa; ^^ ^max]- 

Proof: See Appendix A. ■ 
Now denote the left-hand side (LHS) of ( fT9] l as Ay(f), we 
can rearrange the terms in iT9[ to get: 

Avit)<B+J2 (Enit) - 0„)E{e„(O | Z{t)} (20) 
-E{ Y [VUi'\Rl^\t)) ~ QW(i)i?W(i)] I Z(t)} 

n.c 

-E{EfE E ^^m^^-^^ 

Z{t)}. 



We now present the ESA algorithm. The idea of the algo- 
rithm is to approximately minimize the right-hand side (RHS) 
of (|20] | subject to the energy-availability constraint In the 
algorithm, we use a parameter 7 = Rmax + dmaxl^'max, which 
is used in the link weight definition to allow deterministic 
upper bounds on queue sizes. 

Energy-limited Scheduling Algorithm (ESA): Initialize 6. 
At every slot, observe Q{t), E{t), and S{t), do: 

« Energy Harvesting: At time t, if E„{t) — 9n < 0, perform 
energy harvesting and store the harvested energy, i.e., 
e„(t) ~ hn{t). Else set e„(i) = 0. 

* Data Admission: At every time t, choose Rn^ (t) to be the 
optimal solution of the following optimization problem: 

max : VUi'^^r) ~ Q'^^\t)r, s.t. < r < R^ax- (21) 

• Power Allocation: At every time t, define the weight of 
the commodity c data over link [n, h] as: 

r(c) A \o('^)(t\ _ (0?'=)^ - -^1+ (22) 



Then define the link weight (t) = maxc W^[„ hj (0' 

and choose P{t) £ T"^-'*) to maximize: 



subject to the energy availability constraint 



(23) 



• Routing and Scheduling: For every node n, find any c* G 

argmax,W^£),](i). ^ wffj^it) > 0, set: 

l^lnM^i) = ^'lnMi^)^ (24) 

that is, allocate the full rate over the link [n, b] to any 
commodity that achieves the maximum positive weight 
over the link. Use idle-fill if needed. 

• Queue Update: Update Qn\t) and En{t) according to 
the dynamics (|5]l and (|7]l, respectively. 

Note that ESA only requires the knowledge of the instant 
channel state S{t) and the queue sizes Q{t) and E{t). It does 
not even require any knowledge of the energy state process 
hit). This is very useful in practice when the knowledge 
of the energy source is difficult to obtained. ESA is also 
very different from previous algorithms for energy harvesting 
network, e.g., ||9l ifTol . where statistical knowledge of the 
energy source is often required. Also note that if all the links 
do not interfere with each other, then ESA can easily be 
implemented in a distributed manner, where each node only 
has to know about the queue sizes at its neighbor nodes and 
can decide on the power allocation locally. 

V. Performance Analysis 

We now present the performance results of the ESA algo- 
rithm. In the following, we first present the results under i.i.d. 
network randomness and give its proof in the appendix. We 
later extend the performance results of ESA to the case when 
the network randomness is Markovian. 

A. ESA under I.I.D. randomness 

Here we state the performance of ESA under the case when 
the channel state and the energy state, i.e., S{t) and h{t), are 
both i.i.d. 

Theorem 2: Under the ESA algorithm with 0„ = Sf3V + 
Pmax for all n, we have the following: 

(a) The data queues and the energy queues satisfy the 
following for all time: 

< Q\t\t) <PV + Rmax. V (n, c), (25) 

Oil H" ^max 1 

V n. (26) 

Moreover, when a node n allocates nonzero power to 
any of its outgoing links, En{t) > Pmax- 

(b) Let r = (r"^, V (n, c)) be the time average admitted rate 
vector achieved by ESA, then: 

Utotir) = Ui^Hr^n > Utotir*) ~ |, (27) 

n.c 

where r* is an optimal solution of the UMEM problem, 

and B = B + N^dmaxl^max = independent 
of V. 

Proof: See Appendix B. ■ 
We note the following of Theorem |2] First, we will see 
that Part (a) is indeed a sample path result. Hence it holds 
under arbitrary S{t) and h{t) processes. Thus they also hold 
when S{t) and h{t) evolve according to some finite state 



irreducible and aperiodic Markov chain. Second, by taking 
e = 1/V, we see from Part (a) that the average data queue 
size is 0(l/e). Combining this with Part (b), we see that that 
ESA achieves an [0(e), 0(l/e)] utility-backlog tradeoff for 
the UMEM problem. Third, we see from Part (a) that the 
energy queue size is deterministically upper bounded by some 
0(l/e) constant. This provides an explicit characterization 
of the size of the energy storage device that is needed for 
achieving the desired utility performance. Such explicit bounds 
are particularly useful for system deployments. 

B. ESA under Markovian randomness 

We now extend our results to the more general setting where 
the channel state S{t) and the energy state h{t) both evolve 
according to some finite state irreducible and aperiodic Markov 
chains. Note that in this case Hg. and 71^. represent the steady 
state probability of the events {S{t) = s.;} and {h{t) = hi), 
respectively. In this case, the performance results of ESA are 
summarized in the following theorem: 

Theorem 3: Suppose that S{t) and h{t) evolve according 
to some finite state irreducible and aperiodic Markov chains. 
Then under ESA, we have: (a) the bounds (l25l l and ( |26] | still 
hold, (b) the average utiHty is within 0{1/V) of Utotir*), 
i.e., Utotir) = En.cUn\rn > Utotir*) - Oil/V). 

Proof: Part (a) follows from Theorem|2l since it is indeed 
a sample-path result. The proof of the utility performance is 
similar to that in (24'|, and hence is omitted for brevity. ■ 

VI. Reducing the Buffer size 

In this section, we show that it is possible to achieve 
the same 0(e) close-to-optimal utility performance guarantee 
using energy storage devices with only 0([log(l/e)]^) sizes, 
while guaranteeing a much smaller average data queue size, 
i.e., 0([log(l/e)]^). Our algorithm is motivated by the fol- 
lowing theorem, which is a modified version of Theorem 2 in 
1251 . In the theorem, we denote y = iv,i/). 

Theorem 4: Suppose that hit) and Sit) both evolve ac- 
cording some finite state irreducible and aperiodic Markov 
chain, that y* ~ (r*, u*) is finite and unique, that 6 is chosen 
such that 0„ + f* > 0, V n, and that for all y = iv, u) with 

^ 0,1/ S M^, the dual function giy) satisfies: 

9{y*) > 9{y) + L\\y* ~ y\l (28) 

for some constant L > independent of V . Then y* = QiV), 
and that under ESA, there exists constants D,K,c* ~ 6(1), 
i.e., all independent of V, such that for any m G R+, 

V'-''\D,Km) < c*e-", (29) 

where V'^''\D,K m) is defined: 

1 

P'^''\D,Km) =limsup- VFr{(f(T,m)}, (30) 

t—^QO t 

r=0 

with S'it,m) being the following deviation event: 

^(t,m) = {3in,c),\Ql^\t) -vl^'>*\ > D + Km} (31) 
U{3n, \iEnit)-0n)-K\> D + Km}. 



Proof: The proof is similar in spirit to ||251 and is omitted 
for brevity. ■ 
Note that the finiteness and uniqueness of y* can usually be 
satisfied in practice, particularly when a certain "slackness" 
condition is met. Also note that the condition ( |28] ) can typically 
be satisfied in practice when the action space is finite (See 
1251 for further discussions of these conditions). In this case. 
Theorem Instates that the queue backlog vector pair is "expo- 
nentially attracted" to the fixed point (u*, v* +0) = Q{V), in 
that the probability of deviating decreases exponentially with 
the deviation distance. Therefore, the probability of deviating 
by some Q{\og{V)) distance will be 1/V, which will be very 
small when V is large. This suggests that most of the queue 
backlogs are kept in the queues to maintain a "proper" queue 
vector value to base the decisions on. If we can somehow learn 
the value of this vector, then we can "subtract out" a large 
amount of backlog from the network and reduce the required 
buffer sizes. Below, we present the Modified-ESA (MESA) 
algorithm. 

To start, for a given e, we let V ~ 1/e, and define 
M ~ 4[log(y)]^. We then associate with each node n a virtual 
energy queue process En{t) and a set of virtual data queues 
Qn\t) Vc. We also associate with each node n an actual 
energy queue with size M. We assume that V is chosen to be 
such that ^- > a,nax = max[P„a3;, /imar]- MESA consists of 
two phases: Phase I runs the system using the virtual queue 
processes, to discover the "attraction point" values of the 
queues (as explained below). Phase II then uses these values 
to carefully perform the energy harvesting, power allocation, 
and routing and scheduling actions so as to ensure energy 
availability and reduce network delay. 

Modified-ESA (MESA): Initialize 6. Perform the following: 

• Phase I: Choose a sufficiently large T. From time t = 
0, ...,r, run ESA using Q{t) and E{t) as the data and 
energy queue processes. Obtain the two vectors Q = 

iQn \ V(n,c)) and £ — (f„,Vn) by having: 

Qi;-^ - [QL'Ht) - ^]+, £ = K(r) - (32) 

• Phase II: Reset t = 0. InitiaUze E{0) = £ and 
Q(0) = Q. Also set Q(0) = and E{0) = 0. In 
every time slot, first run the ESA algorithm based on 
Q{t), E{t), and S{t), to obtain the action variables, i.e., 
the corresponding e„(t), Rn\t), and values. 
Perform Data Admisson, Power Allocation, and Routing 
and Scheduling exactly as ESA, plus the following: 

- Energy harvesting: If En{t) < £n, let e,i(<) = 
[en(i) ^ {£n — En{t))]'^ and harvest e{t) amount 
of energy, i.e., update En{t) as follows: 

En{t + l) = {[E^{t)- J2 + AA/. 

Here a Ab = min[a, b]. Else if En{t) > £n + M, do 
not spend any power and update according to: 

En{t + 1) = min [Er,{t) + e„(t), M] . 



Else update En{t) according to: 

En{t + 1)= {[En{t) - Pln,bM'^ + ^"(O) A M. 

- Packet Dropping: For any node n with En{t) < £n + 
Pmax or En [t] > £n + M, drop all the packets that 
should have been transmitted, i.e., change the input 
into any Qn\t) to: 

Here ![.] is the indicator function and Fa{t) is the 
event that i?a(i) e „ + Pmax , 'S' + -^/] ■ Then further 
modify the routing and scheduling action under ESA 
as follows: 

* lfQi.\t) < Q(f\letii^'(<) = [Ai^\t)~[Qi^^- 
Qn\t)]\~^, update Qn\t) according to: 

Qi:Ht + i)<[Qi'Ht)- E /^MiWl^ + ^i'Hi)- 

beA/"^"' 

* If Qii\t) > Q^n \ update Q^n\t) according to: 

- Update E{t) and Q{t) using Q and (|5]l. 

Note here we have used the [•]+ operator for updating En{t) 
in the energy harvesting part. This is due to the fact that the 
power allocation decisions are now made based on E{t) but 
not E{t). Note that if En{t) never gets below f„ or above 
£n + M, then we always have = En{t) —£„■ Similarly, 

if Qn\t) is always above Qn^ and En{t) is always in [£„ + 
Pmax, £n+M], then we always have Q^f^(t) = gi'^(t)-df' . 
Our algorithm is designed to ensure that Q'n \t) and En{t) 
mostly stay in the "right" arrange, as shown in the following 
lemma. 

Lemma 3: For all time t, we have the following: 

< Ql'Ht) < [Qi^\t) Q('=)]+ +7,V(n,c), (33) 
min [[En{t) - f„] + , M] < E^it). (34) 

Proof: See Appendix C. ■ 
We now summarize the performance result of MESA in the 

following theorem: 

Theorem 5: Suppose that the conditions in Theorem |4] 

hold, that the system is in steady state at time T, and that 

a steady state distribution for the queues exists under ESA. 

Then under MESA with a sufficiently large V , with probability 

1 — 0{yz), we have: 

Q < 0{[\og{V)f), (35) 
Utot{r) > Utot{r*)-Oil/V). (36) 

Proof: See Appendix D. ■ 
Note that the conditions in Theorem |4] are indeed the con- 
ditions needed for proving the exponential attraction result in 
[25 1 . Thus Theorem |5] implies that if the exponential attraction 
result holds, which is mostly the case in practice (See ll25l 



for more discussions), then one can significantly reduce the 
energy capacity needed to achieve the 0(e) close-to-optimal 
utility performance and greatly reduce the network congestion. 

VII. Simulation 

In this section we provide simulation results of the ESA 
algorithm. We consider a data collection network shown in 
Fig. [T] Such networks typically appear in the sensor network 
scenario where sensors are used to sense data and forward 
them to the sink. In this network, there are 6 nodes. The nodes 
1,2,3 sense data and deliver them to node Sink via the relay 
of nodes 4, 5. 




Fig. 1. A data collection network, where Li denotes link i. 

The channel state of each communication link Lj, rep- 
resented by a directed edge, can be either "G=Good" or 
"B=Bad", and evolves according to the two-state Markov chain 
shown in Fig. |2] with pa = Pb ^ 0.3. At any time, we can 
allocate either zero or one unit of power. One unit of power 
can serve two packets over a link when the channel state is 
good, but can only serve one when the channel is bad. We 
assume Rmax = 3 and the utility functions are given by: 
Uiir) = U2{r) - Usir) = log(l+r) and Ui{r) = Usir) = 0. 
For simplicity, we also assume that all the links do not interfere 
with each other. 



PG 




Fig. 2. A two-state Markov chain. 

We also assume that for each node, the available energy 
hn{t) evolves according to the same two-state Markov chain in 
Fig.|2l When the state is good, hn{t) = 2, otherwise hn{t) = 
0. It is easy to see that in this case, /3 = 1, J = 2, pmax = 2, 
dmax = 2 and Pmax = 2. Using the results in Theorem |2l we 
set 0n = 5pV + Pmax = 2V + 2. We also see that in this 
case, we can use 7 = dmax l^max + Rmax = 7. The simulation 
results are plotted in Fig. [3] We see in Fig. [3] that the total 
network utility converges quickly to very close to the optimal 
value, which can be shown to be roughly 2.03. We also see 
that the average data queue size and the average energy queue 
size both grow linearly in V. 

Fig. E] also shows two sample-path data queue processes 
and two energy queue processes under V = 100. It can be 
verified that all the queue sizes satisfy the queueing bounds 
in Theorem |2l Interestingly, we see that all the queue sizes 




Fig. 3. Simulation results of ESA. 

are "attracted" to certain fixed points. However, different from 
previous work, e.g., 1251 . we see that the queue size of Qi{t) 
does not approach this fixed point from below. It instead first 
has a "burst" in the early time slots. This is due to the fact 
that the systems "waits" for Ei{t) to come close enough to 
its fixed point. Such an effect can be mitigated by storing an 
initial energy of size 6 in the energy queue. 




Fig. 4. Sample path queue processes. 

We also simulate the MESA algorithm for the same network 
with the same 9 value. We use T = 50V in Phase I for 
obtaining the vectors £ and Q. Fig. |5] plots the performance 
results. We observe that extremely few packets were dropped 
in the simulations (at most 5 out of more than 10^ packets 
were dropped under any V values). The utility again quickly 
converges to the optimal as V increases. We also see from 
the second and third plots that the actual queues only grow 
poly-logarithmically in V, i.e., 0([log(T^)]^), while the virtual 
queues, which are the same as the actual queues under ESA, 
grows linearly in V. This shows a good match between the 
simulation results and Theorem |5] 

VIII. Conclusion 

In this paper, we develop the Energy-limited Scheduling Al- 
gorithm (ESA) for achieving optimal utility in general energy 
harvesting networks equipped with only finite capacity energy 
storage device. We show that ESA is able to achieve an average 
utility that is within 0{e) of the optimal for any e > using 
energy storage devices of 0(l/e) sizes, while guaranteeing 
that the time average network congestion is 0(1/ e). We then 





Fig. 5. Simulation results of MESA. 

also develop the Modified-ESA algorithm (MESA), and show 
that MESA can achieve the same 0(e) utility performance 
using energy storage devices of only 0([log(l/e)]^) sizes. 

Appendix A - Proof of Lemma[2] 

Here we prove Lemma |2] 

Proof: First by squaring both sides of (|5]l, and using the 
fact that for any x G R, ([a;] + )^ < x^, we have: 

[Q(^){t+i)r~[Qi:Ht)r (37) 



By defining B = ^(fi^^^nl^^^ + i?^^^, we see that: 



(38) 



Using a similar approach, we get that: 



(39) 



<& -[E.n{t)-er,][ ^K6](i)-e„(i)], 



where B' — ^{Pmax^nimax 



hm.ax)'^- Now by summing ( |38] | over all 



(n, c) and (l39ll over all n, and by defining B = N^B+NB' = 

N'^{ldl,^x^4,^ax + ^max) + I^l^'max + /imax)^ WC haVC: 

L(t+i)-i(t)<i?-EQ«Ho[ E ^V^) 

- E /^[liW-^i^Ho] 

Taking expectations on both sides over the random channel 
and energy states and the randomness over actions con- 
ditioning on Z{t), subtracting from both sides the term 
VE{J2n.cUn\Rn\t)) \ Z {t)} , and rearranging the terms, 
we see that the lemma follows. ■ 



Appendix B - Proof of Theorem[2] 

Here we prove Theorem |2l The proof idea is as follows: We 
first show that by our choice of 6, the ESA algorithm ensures 
the energy-availability constraint ^ even if we remove it from 
the algorithm. This enables us to show that ESA approximately 
minimizes the value of the RHS of ( |20] | over all possible 
policies. We then analyze the utility performance of ESA by 
relating the value of the RHS of ( l20l i under ESA to the dual 
function gs^,hj{v, i')- 

Proof: (Part (a)) We first prove dZSl ) using a similar 
argument as in fTSl. It is easy to see that it holds for 
t = 0, since Qn\o) — for all (n,c). Now assume that 
Qn\t) < j3V + Rmax for all (n, c) at time t, we want 
to show that it holds for time i + 1. First, if node n does 
not receive any commodity c data from other nodes, then 
Q''n\t) < Qk^\t + 1) < 13V + R^ax- Second, if node n 
receives endogenous commodity c data from any other node 
h. Then according to the ESA algorithm, we must have: 



t\t) 



l<l3V + Rr, 



■ 7- 



However, since any node can receive at most 7 commodity 
c packets, we have QnHt + 1) < I3V + Rmax- Finally, if 
node n receives exogenous packets from outside the network, 
then according to ( |2T] ). we must have Qn\t) < PV. Hence 

Q^n ^t + I) < (3V + Rmax- 

Now it is also easy to see from the energy storage part of 
ESA that En{t) < On + Knax, which proves ( l26l ). 

We now show that if En{t) < Pmax, then G{P{t)) will 
be maximized by choosing P[n,b] (0 = for all b € Afn°^ at 
node n. To see this, first note that since all the actual queues 
are upper bounded by /3V + Rmax, we have: W[n,b]{t) < 
I3V - dmaxfJ-max for all [ji, b] and for all time. 

Now let the power allocation vector that maximizes 
G{P{t)) at time t be P* and assume that there exists some 
■^[n m] ^^^^ positive. We now create a new power allocation 
vector P by setting only P*^ ^^^j = in P* . We see that P 
is also feasible. Then we have the following, in which we 
have written (S'(t), P{t)) only as a function of P{t) to 

simplify notation: 

G{P*)-G{P) 

" fceA/-^"' 

+ {En{t) ^ 6n)P(n,m] 

Here in the last step we have used Q in Property |2] of 

y"[n.m](-,-P), which implies that fi[,„b]{P*) - A*[>i,h](-P) < 
for all b m. Now suppose En{t) < Pmax- We see then 
E„{t) - On < -S/3V. Using Property [T] and the fact that 
W[n^b]{t) < PV ~ dmaxfJ-max, the above impHes: 

G(P*) - G(P) < WV - dmaxt^max)SP{n^m] " S(3VP{nM 

< 0. 



This shows that P* cannot have been the power vector that 
maximizes G{P{t)) if En{t) < Pmax- Therefore En{t) > 



Pmax whenever node n allocates any nonzero power over 
any of its outgoing links. Hence all the power allocation 
decisions are feasible. This shows that the constraint ^ is 
indeed redundant in ESA and completes the proof of Part (a). 

(Part (b)) We now prove Part (b). We first show that 
ESA approximately minimizes the RHS of ( |20| |. To see this, 
note from Part (A) that ESA indeed minimizes the following 
function at time t: 



D{t) = J2iEn{t)-eMt) 



(40) 



[VUi^HRl^Ht))~Qi^\t)Ri^\t)] 



n,c£j\f 

+ {E„{t)-0„) P[nM(t) 

subject to only the constraints; e„(t) G [0,h„{t)], Rl^\t) G 
[0,Rmax], P{t) G V^"''' and i.e., without the energy- 
availability constraint (|4]i. Now define D{t) as follows: 



(41) 



n,c£j\f 

-E [E E ^^';Uim\t)-Qi^\t)\ 



Note that D{t) is indeed the function inside the expectation 
on the RHS of the drift bound ( fT9] l. It is easy to see from the 
above that: 

D{t) = b{t)+YY. E 
Since ESA minimizes D{t), we see that: 

^"w+EE E 4tiW7 



<i)^^^w+EE E 41 



{c)ALT 
b] 



where the superscript E represents the ESA algorithm, and 
ALT represents any other alternate policy. Since 

O^EE E A^M]W7<A^V 



we have: 



D^{t) < D'''^' (t) + N'jdmaxf^r. 



(42) 



That is, the value of D{t) under ESA is no greater than its 
value under any other alternative policy plus a constant. Now 



using the definition of D{t), (fT9T l can be rewritten as: 

A{t) -vEiYutHRi'Ht)) \ Zit)} 

n,c 

<B + E{D^{t) I Z{t)}. 

Using (l42l i. we get: 

Ait) - VE{ Y U,[^HRi'Ht)) I Zit)} (43) 

n,c 

< B + E{D'^^^{t) I Z{t)}, 

where B = B + N'^^dmaxfJ'max- Now consider the policy 
that minimizes D{t) subject to only e„(t) G [0,hn{t)], < 
Rn\t) < Rmax, P G V^^^*^^ and (I3]l, and denote the value 
of D{t) under this policy by D*{t). It is easy to see then 
D*{t) is obtained by minimizing each term in dTTT i over the 
constraints. Hence by comparing D*{t) with ( fTSl l. we see that 
indeed, when S{t) = and h{t) = hj. 

Using this fact in ( |43] |. we have under ESA that: 

A{t) - VE{ Y U,[^HRi'Ht)) I Zit)} (44) 

71, C 

<B-E{gs,,hAQit).0~E{t)) I Zit)}. 

Now using (dSll, i.e., g{v, v) = Y.si '^s, Y.hj '^hj9-n,h, {u, u), 
the above becomes: 



A(t)-yE{^[/(^)(i?(^)(t))|Z(t)} 



(45) 



<B-g{Q{t),e-E{t)). 
By Theorem [T] and Lemma [T] we see that: 

VUtotin <r< 9{v\i^*) < g{Q{t),e - E{t)). 
Plug this into (|45l l, we get: 

A(i)-V^E{^f/(^)(i?i^)(t)) I Z{t)}<B-VUtot{r*). 

n,c 

Taking expectations over Z{t) and summing the above over 
< = 0, T — 1, we have: 

E{i(T) - m} ~vj2HY. u^^i^i^Ht))} 



t=0 n,c 



<TB- TVUtotir*). 

Rearranging the terms, using the facts that L{t) > and 
L(0) = 0, dividing both sides by VT, and taking the liminf 
as T ^ oo, we get: 



1 

liminf Tf;YHll Ui^\Ri^Ht))} > Utot{r*) - B/V. 

t—Q n,c 

Using Jensen's inequality, we see that: 

T-l 



Y f^l^^a^i^f ^ E HRl:\t)}) > Utotir*) - B/V. 

n,c t—0 

This completes the proof of Part (b). 



Appendix C - Proof of Lemma[3] 

Here we prove Lemma |3] 

Proof: We first prove (l33T l. We first define an intermediate 
process Qn\t) that evolves exactly as Qn\t) except that it 
does not discard packets when En{t) < £n+Pviax or £'„(t) > 
£n + M. We see then Qi^\t) < Q^\t). Using Lemma 3 in 
||251. we see that: Q'k\t) < [Q';^\t) - Qi'^'Y + 7- Hence 
Qi"\t) < [Qir\t) - Qi^Y + 7 and (El follows. 

We now look at (|34| |. We see that it holds at time since 
= E„ {0) - £n ~ i^n(O). Now suppose that it holds for 
t = 0, 1, k. We will show that it holds for t = fc + 1. Since 
if £'„(fc + 1) < £n, then (|34] | always holds. Below, we only 
consider the case when iJ„(fc + 1) > f„, i.e.. 



K(fc + 1) - £„]+ = E^[k + 1) - f„ 



(46) 



Also note that since all the actions are made based on 
Q(t) and E(t), by Theorem |2] we always have En(f) > 
^beAA<°' P\n-h\{i), thus; 

K{t + \-)=E,^{i)- J2 + (47) 

We consider the following three cases: 

(I) En{k) < £n- Since En{k + 1) > £n, we must have 
£n — Enik) < e„{k). Then according to the harvesting rule, 

E„{k + 1) 
= min[[i?„(fc)- ^ P[„,6](fc)] + 

+e„(fc)-f„+^„(fc),M] 
>min[[^„(fc)+£;„(fc)- ^ -P[„,6](fc)] + 

+e„(fc) -£:„,M] 
> min [Enik) - ^ P[„.b](fc) + e„(fc) - f„, M] 

= min [En{k + l) - £n,M] 
= min [[£;„(fc + l) -£„] + , A/]. 

Here the first inequality uses the property of [•]+, and 
the second inequality uses En{k) > and En{k) > 

(II) En{k) > £n + M. In this case, we see by the induction 
assumption that En{k) = M. Now by the update rule, we see 
that: 



En{k + 1) = min [£'„(fc) + e„(fc), M] = M. 



(48) 



Thus ([Hi still holds. 

(Ill) £n < En{k) < £n + M- We have two cases: 

(III- A) If En{k + I) - £n < M, then using dUll and 63, 



we have: 

minp„(fc + l)-£„] + ,Af] 
= En{k) - ^ P[„_b](fc) + e„(fc) - f„ 

< min [[[^„(fc) J2 -P[n,b](fc)]++e„(fc),M] 

< min[K(fc)- J2 Pln,b]ik)]+ +en{k),M] 
= En{k + 1). 

Here the first inequality uses the property of the operator [•]+, 
and the second inequahty uses the induction that £'„(fc) > 
min [[E„ik) - £n] + ,M] = [E„{k) - £„] + . 

(III-B) If i^„(fc + l)-£'„ > M, then we must have En{k) > 

£n + M 

o^7nax-! and that E^ik) ^ E^ik) £n ^ Ad ot^nax- 
Using the fact that ^- > Umax, we have: 

^„(fc + l) 

= min[£;„(fc)- P[„,f,](fc) + e„(A:),M] 

> min[£;„(fc)- P[„,b](fc) + e„(/c)-£:„,Af] 

beMi°^ 

= mm[En{k + l)-£n,M], 

which implies Enik + 1) = A/. Thus (O holds. This 
completes the proof of ( l34b and proves the lemma. ■ 

Appendix D - Proof of Theorem |5] 

Here we prove Theorem |5] 

Proof: Since a steady state distribution for the queues 
exists under the ESA algorithm, we see that V^^ \D,Km) is 
the steady state probability that event S'{t, m) happens. Now 
consider a large V value that satisfies ^ = ^[\og{V)]^ > 2D 
and log(X^) > 16K. We have: 



> 



K - K 

By using ( |29] l and the above, we see that 



>41og(y). 



Pr(S{T, ■ 



K 



Using the definition of S{t,m), we see that when V is large 
enough, with probability 1 — 0{1/V^), the vectors E{T) and 
Q{T) satisfy the following for all n, c: 

M 

^n + K)\<^- (49) 



m\T)~v^:^*\<'-^,\En{T) 



M 



Using the fact that Q^"' = [Qi'^(T) - f ]+ and = 
[P„(T) - ^] + , dlSl) and the facts that A/ = 4[log(F)]2 and 
y* = = imply that, when V is large enough, 

with probability 1 - 0(1/V"'^), we have: 

> - v^* > -Mi , V (n, c) s.t. ^.(^)* ^ 0, (50) 
Q(f) VKc)s.t.r;(^)* = 0, (51) 

-^>f„-(e„ + <)>-^,Vn. (52) 



Having established (|50]|-(|52]|. ( |35] ) can now be proven using 
i33[ in Lemma [5] and a same argument as in the proof of 
Theorem 4 in IZSl . 

Now we consider ( |36] ). By Lemma [3] when -E„(t) S 
[£:„+P™ax,^^n+M],wehavei;„(t) > [£'„(t)-f„]+ > P,nax- 
Thus all the power allocations are valid under MESA. Now 
since at every time t, MESA performs ESA's data admis- 
sion, and routing and scheduling actions, if there was no 
packet dropping, then MESA will achieve the same utility 
performance as ESA. However, since all the utility functions 
have bounded derivatives, to prove the utility performance of 
MESA, it suffices to show that the average rate of the packets 
dropped is 0(e) = 0{l/V). 

To prove this, we first see that packet dropping happens at 
time t only when the following event happens, i.e.. 



<f(t) = {3 n,E„{t) < £n+Pmax} 

U{3 n,Enit)>£n + M} 

U{3 {n,c),Qi^\t)<Ql:'>}. 



(53) 



However, assuming (I50b-(l52b hold, the following event must 
happen for (ff{t) to happen: 



U{3in,c),Ql^Ht)<v(^> 



3M 



-} 
}■ 



Therefore (S'{t) C S'{t). However, it is easy to see from (ISTT l 
that i{t) C (^(t,m) with rh = - P„,ax - D)/K ^ 
(|[log(F)]2 - - D)/K. Therefore i{t) C S{t,m). 

Using ( |29] l again, we see that: 

^ t-i ^ t-i 

lim sup — ^Pr(i(r)) < limsup- ^Pr((?(r,m)) 

r— T— 



< c e 



'D)/K 



Using the facts that \ ^og{y)f > 2D and log(y) > 16K, we 



see that: 



2 



- Z) 



5[log(V)]' 



> 



K 

Thus we conclude that: 



K 



> 2oiog(y). 



t-i 

T = 



limsup VPr((#(r)) < 



* P„, 
c e ™ 



^20 



O(l/y20). 



Since at every time slot, the total amount of packets dropped 
is no more than N{ii„iax + Rmax), we see that the average 
rate of packets dropped is 0{\/V). This completes the proof 
of Theorem |5] ■ 
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